{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fHjgY4APuoSS"
      },
      "source": [
        "本文档详细介绍了阿里巴巴团队研发的 Qwen2-VL 系列，这是 Qwen-VL 模型的全面升级版本，旨在提升视觉-语言任务的性能。Qwen2-VL 的核心创新包括引入动态分辨率机制和多模态旋转位置嵌入（M-RoPE）。动态分辨率机制允许模型处理任意分辨率的图像，动态生成视觉 token，并通过压缩邻近 token 减少序列长度，从而高效捕获多尺度信息。而 M-RoPE 将时间、高度和宽度的位置信息结合，显著增强模型对多模态输入（图像、视频和文本）的感知能力。此外，模型通过统一的图像和视频处理框架，结合混合训练策略，进一步提升了对静态和动态内容的理解。\n",
        "\n",
        "模型采用三阶段训练策略：\n",
        "\n",
        " **1.** **第一阶段**：单独训练视觉 Transformer（ViT），重点学习图像与文本之间的语义关系。\n",
        "\n",
        " **2.** **第二阶段**：解冻所有参数，在更广泛的数据集上进行多模态综合训练。\n",
        "\n",
        " **3.** **第三阶段**：冻结 ViT 参数，专注于通过指令数据集微调语言模型，提升其任务执行能力。\n",
        "\n",
        "Qwen2-VL 提供三种规模（2B、8B 和 72B 参数版本），能够适应不同硬件资源和任务需求。其多模态模型框架在视觉问答、文档理解、视频分析和多语言文本识别等任务中表现出色，尤其在处理长视频（超过20分钟）和多语言 OCR 任务时展现了卓越能力。通过分阶段训练和丰富的数据来源，模型实现了在多个视觉语言基准测试中的领先表现，并在任务复杂性和多模态理解的扩展性上设立了新的标杆。\n",
        "\n",
        "Qwen2-VL 系列的关键特性还包括对视频的统一处理（每秒采样两帧，结合 3D 卷积处理帧序列）、动态调整输入分辨率以平衡性能和计算效率，以及对特定场景的功能调用能力优化。整体来看，该模型不仅在技术上推动了视觉语言模型的发展，还通过开源的方式支持了广泛的应用和研究场景，为多模态 AI 系统的未来发展提供了坚实基础。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![jupyter](https://camo.githubusercontent.com/a98a801cab554fd20099a2c3fcce4597f45866b31f5d78f391c91196f4a3e57e/68747470733a2f2f7169616e77656e2d7265732e6f73732d616363656c65726174652d6f766572736561732e616c6979756e63732e636f6d2f5177656e322d564c2f7177656e325f766c5f6672616d65776f726b2e6a7067)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 图片来源:\n",
        "https://github.com/QwenLM/Qwen2-VL"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WgkpnZ3JVrEV"
      },
      "source": [
        "# Qwen2-VL多模态模型代码解析"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mRHQ9RBQWXkn"
      },
      "source": [
        "## **1.模型主要模块**\n",
        "Qwen2-VL 模型的核心模块包括 **视觉编码器**、**语言模型** 和 **多模态融合**，它们分别处理图像和文本数据，并结合两者的特征以完成任务。下面将详细介绍这些模块，并结合具体代码进行解析。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TiTWbxSJSHIW"
      },
      "source": [
        "## **1.1 核心模块功能详解**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MwK19i7HSt9X"
      },
      "source": [
        "## 1.1.1 _get_unpad_data\n",
        "\n",
        "**功能：** 用于处理批量序列数据的填充与解填充：\n",
        "\n",
        "**输入：**attention_mask，标识序列中有效元素的二进制掩码。\n",
        "\n",
        "**输出：**\n",
        "\n",
        "  **有效索引：**去掉padding后的序列索引。\n",
        "\n",
        "  **累计长度（cu_seqlens）：** 分隔不同序列的位置。\n",
        "\n",
        "  **最大长度：** 为高效计算确定所需的最大序列长度。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1d_aeJakSH7l"
      },
      "outputs": [],
      "source": [
        "def _get_unpad_data(attention_mask):\n",
        "    seqlens_in_batch = attention_mask.sum(axis=-1, dtype=\"int32\")  # 计算每个序列的长度（非0的元素数量）。\n",
        "    indices = paddle.nonzero(attention_mask.flatten(), as_tuple=False).flatten()  # 找出mask中非0元素的索引。\n",
        "    max_seqlen_in_batch = seqlens_in_batch.max().item()  # 找出batch中最大序列长度。\n",
        "    cu_seqlens = F.pad(paddle.cumsum(seqlens_in_batch, axis=0), (1, 0))  # 累加每个序列长度，用于unpadding。\n",
        "    return (indices, cu_seqlens, max_seqlen_in_batch)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9L1VI8kMTZK-"
      },
      "source": [
        "### 1.1.2    is_casual_mask\n",
        "\n",
        "**功能：** 判断一个掩码矩阵是否是因果掩码（Causal Mask）：\n",
        "\n",
        "因果掩码的特点是**只允许访问当前位置及之前的序列**，因此是一个**上三角矩阵**。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EhxqP-mBTeVl"
      },
      "outputs": [],
      "source": [
        "def is_casual_mask(attention_mask):\n",
        "    return (paddle.triu(attention_mask) == attention_mask).all().item()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xvmDgfloTspM"
      },
      "source": [
        "### 1.1.3 _make_causal_mask\n",
        "\n",
        "**功能：** 创建**因果掩码**，并支持增量注意力机制：\n",
        "\n",
        "适用于自回归模型的注意力机制（防止未来时刻的信息泄露）。\n",
        "支持结合过去的注意力历史（past_key_values_length），扩展到更大的维度。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Whr13NsuTjJi"
      },
      "outputs": [],
      "source": [
        "def _make_causal_mask(input_ids_shape, past_key_values_length):\n",
        "    batch_size, target_length = input_ids_shape\n",
        "    mask = paddle.tril(paddle.ones((target_length, target_length), dtype=\"bool\"))  # 创建下三角矩阵。\n",
        "    if past_key_values_length > 0:\n",
        "        mask = paddle.concat([paddle.ones([target_length, past_key_values_length], dtype=\"bool\"), mask], axis=-1)\n",
        "    return mask[None, None, :, :].expand([batch_size, 1, target_length, target_length + past_key_values_length])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "86WvCxnTUAYP"
      },
      "source": [
        "### 1.1.4 _expand_2d_mask\n",
        "\n",
        "**功能：** 将一个简单的二维掩码扩展为适合注意力计算的4维掩码：\n",
        "\n",
        "输入：[batch_size, src_length]。\n",
        "输出：[batch_size, 1, tgt_length, src_length]，为注意力模块准备的输入。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5WtoBd3wUIjH"
      },
      "outputs": [],
      "source": [
        "def _expand_2d_mask(mask, dtype, tgt_length):\n",
        "    batch_size, src_length = mask.shape[0], mask.shape[-1]\n",
        "    tgt_length = tgt_length if tgt_length is not None else src_length\n",
        "    mask = mask[:, None, None, :].astype(\"bool\")  # 增加维度，适配注意力计算。\n",
        "    mask.stop_gradient = True\n",
        "    expanded_mask = mask.expand([batch_size, 1, tgt_length, src_length])\n",
        "    return expanded_mask\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FpNaAqGzVBE3"
      },
      "source": [
        "### 1.1.5 Qwen2VLCausalLMOutputWithPast\n",
        "\n",
        "这是一个输出类，专为 Qwen2-VL 模型生成语言模型输出设计，继承了 ModelOutput 基类，包含以下字段：\n",
        "\n",
        "**loss:** 可选的语言建模损失，用于训练时评估预测的准确性。\n",
        "\n",
        "**logits:** 模型生成的预测分数，维度为 (batch_size, sequence_length, vocab_size)，表示每个时间步的词汇预测分数。\n",
        "\n",
        "**past_key_values:** 用于缓存历史计算的注意力 key 和 value，可加速序列生成。\n",
        "\n",
        "**hidden_states:** 存储每一层的隐藏状态，可用于进一步分析模型内部表示。\n",
        "\n",
        "**attentions:** 存储注意力权重，用于理解模型的注意力分配。\n",
        "\n",
        "**rope_deltas:** 表示序列长度和多模态旋转位置编码之间的索引差值。\n",
        "\n",
        "**用途：**为 Qwen2-VL 模型在推理和训练阶段返回丰富的信息。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e6oJdKaUVHSM"
      },
      "outputs": [],
      "source": [
        "class Qwen2VLCausalLMOutputWithPast(ModelOutput):\n",
        "    \"\"\"\n",
        "    Base class for Qwen2VL causal language model (or autoregressive) outputs.\n",
        "\n",
        "    Args:\n",
        "        loss (`paddle.Tensor` of shape `(1,)`, *optional*, returned when `labels` is provided):\n",
        "            Language modeling loss (for next-token prediction).\n",
        "        logits (`paddle.Tensor` of shape `(batch_size, sequence_length, config.vocab_size)`):\n",
        "            Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).\n",
        "        past_key_values (`tuple(tuple(paddle.Tensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`):\n",
        "            Tuple of `tuple(paddle.Tensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape\n",
        "            `(batch_size, num_heads, sequence_length, embed_size_per_head)`)\n",
        "\n",
        "            Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see\n",
        "            `past_key_values` input) to speed up sequential decoding.\n",
        "        hidden_states (`tuple(paddle.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`):\n",
        "            Tuple of `paddle.Tensor` (one for the output of the embeddings, if the model has an embedding layer, +\n",
        "            one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.\n",
        "\n",
        "            Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.\n",
        "        attentions (`tuple(paddle.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`):\n",
        "            Tuple of `paddle.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length,\n",
        "            sequence_length)`.\n",
        "\n",
        "            Attentions weights after the attention softmax, used to compute the weighted average in the self-attention\n",
        "            heads.\n",
        "        rope_deltas (`paddle.Tensor` of shape `(batch_size, )`, *optional*):\n",
        "            The rope index difference between sequence length and multimodal rope.\n",
        "    \"\"\"\n",
        "\n",
        "    loss: Optional[paddle.Tensor] = None\n",
        "    logits: paddle.Tensor = None\n",
        "    past_key_values: Optional[List[paddle.Tensor]] = None\n",
        "    hidden_states: Optional[Tuple[paddle.Tensor]] = None\n",
        "    attentions: Optional[Tuple[paddle.Tensor]] = None\n",
        "    rope_deltas: Optional[paddle.Tensor] = None"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-1aPcX3XVupt"
      },
      "source": [
        "### 1.1.6 Qwen2RotaryEmbedding\n",
        "\n",
        "**Qwen2RotaryEmbedding**\n",
        "\n",
        "这是一个实现旋转位置编码（Rotary Position Embedding, RoPE）的模块：\n",
        "\n",
        "__init__:\n",
        "初始化时，基于 dim（嵌入维度）和最大序列长度，计算 inv_freq（逆频率）。\n",
        "使用逆频率和位置索引生成预计算的正弦 (cos_cached) 和余弦 (sin_cached) 编码，节约计算时间。\n",
        "\n",
        "**_set_cos_sin_cache:**\n",
        "动态更新缓存，确保支持超出预定义长度的序列。\n",
        "\n",
        "**forward:**\n",
        "根据输入 x 和序列长度 seq_len 返回对应的正弦和余弦值，以供旋转位置编码使用。\n",
        "\n",
        "**作用：** RoPE 是一种位置编码方法，允许模型通过对查询和键嵌入进行旋转操作，注入序列中的位置信息，尤其适合 Transformer 架构"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "p9q4PBZMWD3e"
      },
      "outputs": [],
      "source": [
        "class Qwen2RotaryEmbedding(nn.Layer):\n",
        "    def __init__(self, dim, max_position_embeddings=2048, base=10000):\n",
        "        super().__init__()\n",
        "        self.dim = dim\n",
        "        self.max_position_embeddings = max_position_embeddings\n",
        "        self.base = base\n",
        "        # [dim / 2]\n",
        "        self.inv_freq = 1.0 / (self.base ** (paddle.cast(paddle.arange(0, self.dim, 2), dtype=\"float32\") / self.dim))\n",
        "        self._set_cos_sin_cache(seq_len=max_position_embeddings)\n",
        "\n",
        "    def _set_cos_sin_cache(self, seq_len):\n",
        "        self.max_seq_len_cached = seq_len\n",
        "        # [seq_len]\n",
        "        t = paddle.arange(seq_len, dtype=\"float32\")\n",
        "        # [seq_len, dim/2]\n",
        "        freqs = paddle.einsum(\"i,j->ij\", t, self.inv_freq)\n",
        "        # Different from paper, but it uses a different permutation in order to obtain the same calculation\n",
        "        # [seq_len, dim]\n",
        "        emb = paddle.concat([freqs, freqs], axis=-1)\n",
        "        # [1, seqlen, 1, dim]\n",
        "        self.cos_cached = emb.cos()\n",
        "        self.sin_cached = emb.sin()\n",
        "\n",
        "    def forward(self, x, seq_len=None):\n",
        "        # x: [bs, num_attention_heads, seq_len, head_size]\n",
        "        if seq_len > self.max_seq_len_cached:\n",
        "            self._set_cos_sin_cache(seq_len)\n",
        "        cos = self.cos_cached[:seq_len]\n",
        "        sin = self.sin_cached[:seq_len]\n",
        "        return (\n",
        "            cos.cast(x.dtype) if cos.dtype != x.dtype else cos,\n",
        "            sin.cast(x.dtype) if sin.dtype != x.dtype else sin,\n",
        "        )\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nxpMwnxFPVgW"
      },
      "source": [
        "# 统一视觉处理方式\n",
        "Qwen2VL采用混合训练方案，结合图像和视频数据，确保图像理解和视频理解的熟练程度。\n",
        "\n",
        "为了尽可能完整地保存视频信息，Qwen2-VL以每秒两帧的频率对每个视频进行采样。\n",
        "\n",
        "集成了深度为2的3D卷积来处理视频输入，使模型能够处理3D tubes 而不是2D patches，从而使其能够在不增加序列长度的情况下处理更多的视频帧。\n",
        "\n",
        "为了保持一致性，每个图像都被视为两个相同的帧。\n",
        "\n",
        "为了平衡长视频处理的计算需求和整体训练效率，我们动态调整每个视频帧的分辨率，将每个视频的token总数限制在16384。这种训练方法在模型理解长视频的能力和训练效率之间取得了平衡。\n",
        "\n",
        "\n",
        "                            "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ejsZsgeRPwJZ"
      },
      "source": [
        "# 原生动态分辨率处理\n",
        "Qwen2 VL的一个关键架构改进是引入了原生动态分辨率支持。与Qwen-VL不同，Qwen2-VL可以处理任意分辨率的图像，将其动态转换为可变数量的视觉标记。\n",
        "\n",
        "为了支持这一功能，Qwen2-VL修改了ViT，删除了原始的绝对位置嵌入，并引入了**2D RoPE**来捕获图像的二维位置信息。\n",
        "\n",
        "在推理阶段，不同分辨率的图像被打包成一个序列，打包长度受到控制以限制GPU内存的使用。\n",
        "\n",
        "此外，为了减少每个图像的视觉标记，在ViT之后使用一个简单的MLP层将相邻的2×2标记压缩成一个标记，并在压缩的视觉标记的开头和结尾放置特殊的**<|vision_start|>**和**<|visition_end|>**标记。\n",
        "\n",
        "**这一版本的Qwen2-VL并没有采用当下流行的大图切分方式**（比如LLava-Next，InternVL2.5，以及MiniCPM-V），而是直接对图像进行**patch**化，然后直接过image encoder进行特征提取，最后对齐到LLM之前，使用**PatchMerger**层进行视觉token数的压缩与进一步提取特征（两层MLP）。\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fgNyHohKPuj_"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BQ1Cm_lTXIAh"
      },
      "source": [
        "### 1.1.7 VisionRotaryEmbedding\n",
        "\n",
        "**VisionRotaryEmbedding**\n",
        "\n",
        "**功能：** 用于计算视觉数据的旋转位置嵌入（Rotary Position Embedding, RPE）。\n",
        "\n",
        "**具体功能：**\n",
        "初始化时根据给定的维度 dim 和比例参数 theta 计算逆频率（inv_freq），用于生成不同维度的旋转嵌入频率。\n",
        "\n",
        "在 **forward**方法中，基于序列长度 seqlen 和逆频率生成频率矩阵 freqs。\n",
        "\n",
        "**作用：**\n",
        "生成与视觉数据位置相关的嵌入信息，用于将位置信息注入到视觉特征中。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fJDsqzQIXgvK"
      },
      "outputs": [],
      "source": [
        "class VisionRotaryEmbedding(nn.Layer):\n",
        "    def __init__(self, dim: int, theta: float = 10000.0) -> None:\n",
        "        super().__init__()\n",
        "        self.inv_freq = 1.0 / theta ** (paddle.arange(start=0, end=dim, step=2, dtype=\"float32\") / dim)\n",
        "\n",
        "    def forward(self, seqlen: int) -> paddle.Tensor:\n",
        "        seq = paddle.arange(seqlen).cast(self.inv_freq.dtype)\n",
        "        freqs = paddle.outer(x=seq, y=self.inv_freq)\n",
        "        return freqs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l_TYlUaYXnGQ"
      },
      "source": [
        "### 1.1.8 PatchEmbed\n",
        "\n",
        "**PatchEmbed**\n",
        "\n",
        "**功能：** 将输入的图像或视频数据分割成小块（Patch），并通过卷积操作提取特征，生成低维嵌入表示。\n",
        "\n",
        "**具体功能：**\n",
        "初始化时定义了一个 3D 卷积层（Conv3D），其卷积核大小由 patch_size（空间维度）和 temporal_patch_size（时间维度）决定，用于提取时空特征。\n",
        "\n",
        "在 forward 方法中，将输入的视觉数据重新调整为适合 3D 卷积的形状，然后通过卷积层投影到低维特征空间。\n",
        "\n",
        "针对特定硬件（如 NPU），增加了支持 FP16 或 BF16 数据类型的特定处理逻辑。\n",
        "\n",
        "**作用：**\n",
        "将高维的视觉数据（图像或视频帧）映射为低维特征嵌入，作为后续处理的输入。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "n5Gdh0GaX1WY"
      },
      "outputs": [],
      "source": [
        "class PatchEmbed(nn.Layer):\n",
        "    def __init__(\n",
        "        self,\n",
        "        patch_size: int = 14,\n",
        "        temporal_patch_size: int = 2,\n",
        "        in_channels: int = 3,\n",
        "        embed_dim: int = 1152,\n",
        "    ) -> None:\n",
        "        super().__init__()\n",
        "        self.patch_size = patch_size\n",
        "        self.temporal_patch_size = temporal_patch_size\n",
        "        self.in_channels = in_channels\n",
        "        self.embed_dim = embed_dim\n",
        "\n",
        "        kernel_size = [temporal_patch_size, patch_size, patch_size]\n",
        "        self.proj = nn.Conv3D(in_channels, embed_dim, kernel_size=kernel_size, stride=kernel_size, bias_attr=False)\n",
        "\n",
        "    def forward(self, hidden_states: paddle.Tensor) -> paddle.Tensor:\n",
        "\n",
        "        target_dtype = self.proj.weight.dtype\n",
        "        hidden_states = hidden_states.reshape(\n",
        "            [-1, self.in_channels, self.temporal_patch_size, self.patch_size, self.patch_size]\n",
        "        )\n",
        "\n",
        "        if _IS_NPU:\n",
        "            # NOTE: In npu device, conv3d only support fp16 or bf16 dtype.\n",
        "            hidden_states = F.conv3d(\n",
        "                hidden_states.cast(paddle.bfloat16),\n",
        "                self.proj.weight.cast(paddle.bfloat16),\n",
        "                stride=self.proj._stride)\n",
        "            hidden_states = hidden_states.to(target_dtype).reshape([-1, self.embed_dim])\n",
        "        else:\n",
        "            hidden_states = self.proj(hidden_states.to(dtype=target_dtype)).reshape([-1, self.embed_dim])\n",
        "        return hidden_states"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2kuoUCSmX-NP"
      },
      "source": [
        "### 1.1.9 PatchMerger\n",
        "\n",
        "**PatchMerger**\n",
        "\n",
        "**功能：** 合并视觉特征块（Patches），通过融合邻近特征块的信息，生成新的特征表示。\n",
        "\n",
        "**具体功能：**\n",
        "初始化时：\n",
        "定义了一个 LayerNorm 层（ln_q）进行特征的标准化。\n",
        "\n",
        "定义了一个简单的全连接层（MLP）用于非线性变换。\n",
        "\n",
        "**在 forward 方法中：**\n",
        "对输入特征进行标准化，并调整形状后通过 MLP 进行特征融合。\n",
        "\n",
        "**作用：**\n",
        "通过特征标准化和非线性变换，增强特征的表达能力。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZEKYBQE4YUWa"
      },
      "outputs": [],
      "source": [
        "class PatchMerger(nn.Layer):\n",
        "    def __init__(self, dim: int, context_dim: int, spatial_merge_size: int = 2) -> None:\n",
        "        super().__init__()\n",
        "        self.hidden_size = context_dim * (spatial_merge_size**2)\n",
        "        self.ln_q = nn.LayerNorm(context_dim, epsilon=1e-6)\n",
        "        self.mlp = nn.Sequential(\n",
        "            nn.Linear(self.hidden_size, self.hidden_size),\n",
        "            nn.GELU(),\n",
        "            nn.Linear(self.hidden_size, dim),\n",
        "        )\n",
        "\n",
        "    def forward(self, x: paddle.Tensor) -> paddle.Tensor:\n",
        "        x = self.mlp(self.ln_q(x).reshape([-1, self.hidden_size]))\n",
        "        return x"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7zy6DD13YvJq"
      },
      "source": [
        "### 1.1.10 VisionMlp\n",
        "\n",
        "**VisionMlp**\n",
        "\n",
        "**功能：** 一个经典的多层感知机（MLP），用于对视觉特征进行进一步的非线性变换。\n",
        "\n",
        "**具体功能：**\n",
        "初始化时：\n",
        "定义了两层全连接层（fc1 和 fc2）。\n",
        "使用 hidden_act 指定的激活函数（如 ReLU 或 GELU）进行非线性变换。\n",
        "\n",
        "**在 forward 方法中：**\n",
        "将输入依次通过 fc1、激活函数、fc2，生成新的特征表示。\n",
        "\n",
        "**作用：**\n",
        "对视觉特征进行非线性处理，提升特征的表达能力，适配后续任务。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LxbmfHcdZfXE"
      },
      "outputs": [],
      "source": [
        "class VisionMlp(nn.Layer):\n",
        "    def __init__(self, dim: int, hidden_dim: int, hidden_act: str) -> None:\n",
        "        super().__init__()\n",
        "        self.fc1 = nn.Linear(dim, hidden_dim)\n",
        "        self.act = ACT2FN[hidden_act]\n",
        "        self.fc2 = nn.Linear(hidden_dim, dim)\n",
        "\n",
        "    def forward(self, x) -> paddle.Tensor:\n",
        "        return self.fc2(self.act(self.fc1(x)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pCU2T4l2ZjZq"
      },
      "source": [
        "### 1.1.11 VisionAttention\n",
        "\n",
        "**VisionAttention**\n",
        "\n",
        "**初始化方法**\n",
        "\n",
        "**dim:** 输入特征的维度。\n",
        "\n",
        "**num_heads:** 注意力头的数量。\n",
        "\n",
        "计算 head_dim，即每个头分配的维度大小（dim // num_heads）。\n",
        "\n",
        "\n",
        "**前向传播方法：**\n",
        "\n",
        "**输入：**\n",
        "**hidden_states:** 输入的特征张量，形状为 (seq_length, dim)。\n",
        "\n",
        "**cu_seqlens:** 表示累积序列长度，用于区分批次的序列范围。\n",
        "\n",
        "**rotary_pos_emb:** 旋转位置嵌入，用于增强位置编码的能力。\n",
        "\n",
        "**过程：**\n",
        "\n",
        "使用 qkv 全连接层将输入分解为查询 (Q)、键 (K)、值 (V)。\n",
        "\n",
        "使用 apply_rotary_pos_emb_vision 应用旋转位置嵌入。\n",
        "\n",
        "构造注意力掩码，用于屏蔽不相关的位置。\n",
        "\n",
        "计算注意力分数 (attn_weights) 并应用 softmax。\n",
        "\n",
        "根据权重聚合值 (V) 得到注意力输出。\n",
        "\n",
        "使用 proj 层投影输出。\n",
        "\n",
        "**输出：**\n",
        "注意力模块的输出特征。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9FpG1t6ZaJoY"
      },
      "outputs": [],
      "source": [
        "class VisionAttention(nn.Layer):\n",
        "    def __init__(self, dim: int, num_heads: int = 16) -> None:\n",
        "        super().__init__()\n",
        "        self.num_heads = num_heads\n",
        "        self.qkv = nn.Linear(dim, dim * 3, bias_attr=True)\n",
        "        self.proj = nn.Linear(dim, dim)\n",
        "        self.head_dim = dim // num_heads  # must added\n",
        "\n",
        "    def forward(\n",
        "        self, hidden_states: paddle.Tensor, cu_seqlens: paddle.Tensor, rotary_pos_emb: paddle.Tensor = None\n",
        "    ) -> paddle.Tensor:\n",
        "        seq_length = hidden_states.shape[0]\n",
        "        q, k, v = (\n",
        "            self.qkv(hidden_states).reshape([seq_length, 3, self.num_heads, -1]).transpose([1, 0, 2, 3]).unbind(0)\n",
        "        )\n",
        "        q = apply_rotary_pos_emb_vision(q.unsqueeze(0), rotary_pos_emb).squeeze(0)\n",
        "        k = apply_rotary_pos_emb_vision(k.unsqueeze(0), rotary_pos_emb).squeeze(0)\n",
        "\n",
        "        attention_mask = paddle.zeros([1, seq_length, seq_length], dtype=\"bool\")\n",
        "        for i in range(1, len(cu_seqlens)):\n",
        "            attention_mask[..., cu_seqlens[i - 1] : cu_seqlens[i], cu_seqlens[i - 1] : cu_seqlens[i]] = True\n",
        "\n",
        "        zero = paddle.zeros(attention_mask.shape, dtype=hidden_states.dtype)\n",
        "        neg_inf = paddle.full_like(attention_mask, paddle.finfo(hidden_states.dtype).min, dtype=hidden_states.dtype)\n",
        "        attention_mask = paddle.where(attention_mask, zero, neg_inf)\n",
        "\n",
        "        q = q.transpose([1, 0, 2])\n",
        "        k = k.transpose([1, 0, 2])\n",
        "        v = v.transpose([1, 0, 2])\n",
        "        attn_weights = paddle.matmul(q, k.transpose([0, 2, 1])) / math.sqrt(self.head_dim)\n",
        "        attn_weights = attn_weights + attention_mask\n",
        "        attn_weights = nn.functional.softmax(attn_weights, axis=-1, dtype=\"float32\")\n",
        "        attn_output = paddle.matmul(attn_weights, v)\n",
        "        attn_output = attn_output.transpose([1, 0, 2])\n",
        "        attn_output = attn_output.reshape([seq_length, -1])\n",
        "        attn_output = self.proj(attn_output)\n",
        "        return attn_output\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kzzb9GkwaNwg"
      },
      "source": [
        "### 1.1.12 VisionFlashAttention2\n",
        "\n",
        "**VisionFlashAttention2**\n",
        "\n",
        "这是一个优化版本的注意力实现，利用了**Flash Attention**的快速计算特性。\n",
        "\n",
        "**与 VisionAttention 的区别：**\n",
        "\n",
        "利用 flash_attn_varlen_func 优化了注意力分数的计算。\n",
        "直接处理长序列的注意力计算，提升速度和效率。\n",
        "所有计算在 bfloat16 类型下进行，最后转回 float32。\n",
        "\n",
        "**新增参数：**\n",
        "softmax_scale: 手动加上的缩放因子（head_dim**-0.5）。\n",
        "\n",
        "**改进：**\n",
        "支持变长序列的高效计算。\n",
        "更适合处理大规模视觉任务。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QFgSz4VHavHx"
      },
      "outputs": [],
      "source": [
        "class VisionFlashAttention2(nn.Layer):\n",
        "    def __init__(self, dim: int, num_heads: int = 16) -> None:\n",
        "        super().__init__()\n",
        "        self.num_heads = num_heads\n",
        "        self.qkv = nn.Linear(dim, dim * 3, bias_attr=True)\n",
        "        self.proj = nn.Linear(dim, dim)\n",
        "        self.head_dim = dim // num_heads  # must added\n",
        "\n",
        "    def forward(\n",
        "        self, hidden_states: paddle.Tensor, cu_seqlens: paddle.Tensor, rotary_pos_emb: paddle.Tensor = None\n",
        "    ) -> paddle.Tensor:\n",
        "        seq_length = tuple(hidden_states.shape)[0]\n",
        "        qkv = self.qkv(hidden_states).reshape([seq_length, 3, self.num_heads, -1]).transpose(perm=[1, 0, 2, 3])\n",
        "        q, k, v = qkv.unbind(axis=0)\n",
        "        q = apply_rotary_pos_emb_vision(q.unsqueeze(axis=0), rotary_pos_emb).squeeze(axis=0)\n",
        "        k = apply_rotary_pos_emb_vision(k.unsqueeze(axis=0), rotary_pos_emb).squeeze(axis=0)\n",
        "        max_seqlen = (cu_seqlens[1:] - cu_seqlens[:-1]).max().item()\n",
        "\n",
        "        softmax_scale = self.head_dim**-0.5  # TODO: 需要手动加上\n",
        "        attn_output = (\n",
        "            flash_attn_varlen_func(  # flash_attn_unpadded\n",
        "                q.astype(\"bfloat16\"),  # 不支持float32\n",
        "                k.astype(\"bfloat16\"),\n",
        "                v.astype(\"bfloat16\"),\n",
        "                cu_seqlens,\n",
        "                cu_seqlens,\n",
        "                max_seqlen,\n",
        "                max_seqlen,\n",
        "                scale=softmax_scale,  # TODO: 需要手动加上\n",
        "            )[0]\n",
        "            .squeeze(0)\n",
        "            .reshape([seq_length, -1])\n",
        "        )\n",
        "        attn_output = attn_output.astype(paddle.float32)\n",
        "        attn_output = self.proj(attn_output)\n",
        "        return attn_output\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "46dkPe_4a0YA"
      },
      "source": [
        "### 1.1.12 Qwen2VLVisionBlock\n",
        "\n",
        "**Qwen2VLVisionBlock**\n",
        "\n",
        "视觉模型的基本块，包含注意力和前馈网络。\n",
        "\n",
        "**结构：**\n",
        "\n",
        "包含两个子模块：\n",
        "注意力模块 (self.attn)。\n",
        "前馈网络 (self.mlp)。\n",
        "\n",
        "两个子模块之间通过残差连接和层归一化进行信息传递。\n",
        "\n",
        "**前向传播方法：**\n",
        "\n",
        "对输入依次应用注意力模块和前馈网络。\n",
        "每个模块都包含残差连接，防止梯度消失。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "990fStSFcB3Q"
      },
      "outputs": [],
      "source": [
        "class Qwen2VLVisionBlock(nn.Layer):\n",
        "    def __init__(self, config, attn_implementation: str = \"sdpa\") -> None:\n",
        "        super().__init__()\n",
        "        self.norm1 = nn.LayerNorm(config.embed_dim, epsilon=1e-6)\n",
        "        self.norm2 = nn.LayerNorm(config.embed_dim, epsilon=1e-6)\n",
        "        mlp_hidden_dim = int(config.embed_dim * config.mlp_ratio)\n",
        "\n",
        "        self.attn = create_attention_module(config, \"vision\")\n",
        "        self.mlp = VisionMlp(dim=config.embed_dim, hidden_dim=mlp_hidden_dim, hidden_act=config.hidden_act)\n",
        "\n",
        "    def forward(self, hidden_states, cu_seqlens, rotary_pos_emb) -> paddle.Tensor:\n",
        "        hidden_states = hidden_states + self.attn(\n",
        "            self.norm1(hidden_states), cu_seqlens=cu_seqlens, rotary_pos_emb=rotary_pos_emb\n",
        "        )\n",
        "        hidden_states = hidden_states + self.mlp(self.norm2(hidden_states))\n",
        "        return hidden_states"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1aYB39o6cJQc"
      },
      "source": [
        "### 1.1.13 _prepare_4d_causal_attention_mask_with_cache_position\n",
        "\n",
        "**_prepare_4d_causal_attention_mask_with_cache_position**\n",
        "\n",
        "生成一个4D的因果掩码，用于控制注意力机制中的信息流。\n",
        "\n",
        "**功能**\n",
        "\n",
        "将二维的注意力掩码扩展为四维。\n",
        "\n",
        "支持生成过程中静态缓存的处理。\n",
        "\n",
        "**主要步骤**\n",
        "\n",
        "如果输入已经是四维，直接返回。\n",
        "否则，创建一个上三角矩阵，确保注意力只能流向过去或当前时间步。\n",
        "根据 cache_position 和输入掩码动态调整矩阵。\n",
        "扩展为 [batch_size, 1, query_length, key_value_length] 的形状。\n",
        "\n",
        "**关键参数**\n",
        "\n",
        "**min_dtype:** 表示数值的最小值（负无穷），用于屏蔽注意力。\n",
        "\n",
        "**cache_position:** 用于跟踪当前生成的位置。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WqXycytZcbtt"
      },
      "outputs": [],
      "source": [
        "def _prepare_4d_causal_attention_mask_with_cache_position(\n",
        "    attention_mask: paddle.Tensor,\n",
        "    sequence_length: int,\n",
        "    target_length: int,\n",
        "    dtype: paddle.dtype,\n",
        "    min_dtype: float,\n",
        "    cache_position: paddle.Tensor,\n",
        "    batch_size: int,\n",
        "):\n",
        "    \"\"\"\n",
        "    Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape\n",
        "    `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing.\n",
        "\n",
        "    Args:\n",
        "        attention_mask (`paddle.Tensor`):\n",
        "            A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`.\n",
        "        sequence_length (`int`):\n",
        "            The sequence length being processed.\n",
        "        target_length (`int`):\n",
        "            The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet.\n",
        "        dtype (`paddle.dtype`):\n",
        "            The dtype to use for the 4D attention mask.\n",
        "        min_dtype (`float`):\n",
        "            The minimum value representable with the dtype `dtype`.\n",
        "        cache_position (`paddle.Tensor`):\n",
        "            Indices depicting the position of the input sequence tokens in the sequence.\n",
        "        batch_size (`paddle.Tensor`):\n",
        "            Batch size.\n",
        "    \"\"\"\n",
        "    if attention_mask is not None and attention_mask.dim() == 4:\n",
        "        # In this case we assume that the mask comes already in inverted form and requires no inversion or slicing.\n",
        "        causal_mask = attention_mask\n",
        "    else:\n",
        "        causal_mask = paddle.full([sequence_length, target_length], fill_value=min_dtype, dtype=dtype)\n",
        "        if sequence_length != 1:\n",
        "            causal_mask = paddle.triu(x=causal_mask, diagonal=1)\n",
        "        causal_mask *= paddle.arange(target_length) > cache_position.reshape([-1, 1])\n",
        "        causal_mask = causal_mask[None, None, :, :].expand(shape=[batch_size, 1, -1, -1])\n",
        "        if attention_mask is not None:\n",
        "            causal_mask = causal_mask.clone()\n",
        "            mask_length = tuple(attention_mask.shape)[-1]\n",
        "            padding_mask = causal_mask[:, :, :, :mask_length] + attention_mask[:, None, None, :]\n",
        "            padding_mask = padding_mask == 0\n",
        "            causal_mask[:, :, :, :mask_length] = causal_mask[:, :, :, :mask_length].masked_fill(\n",
        "                mask=padding_mask, value=min_dtype\n",
        "            )\n",
        "\n",
        "    return causal_mask"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XDIwmx4XsYzG"
      },
      "source": [
        "### 1.1.14 Qwen2VLVisionBlock\n",
        "\n",
        "**Qwen2VLVisionBlock**\n",
        "\n",
        "视觉模型的基本块，包含注意力和前馈网络。\n",
        "\n",
        "**结构：**\n",
        "\n",
        "包含两个子模块：\n",
        "注意力模块 (self.attn)。\n",
        "前馈网络 (self.mlp)。\n",
        "\n",
        "两个子模块之间通过残差连接和层归一化进行信息传递。\n",
        "\n",
        "**前向传播方法：**\n",
        "\n",
        "对输入依次应用注意力模块和前馈网络。\n",
        "每个模块都包含残差连接，防止梯度消失。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mq8Ka37-sdg4"
      },
      "outputs": [],
      "source": [
        "class Qwen2VLVisionBlock(nn.Layer):\n",
        "    def __init__(self, config, attn_implementation: str = \"sdpa\") -> None:\n",
        "        super().__init__()\n",
        "        self.norm1 = nn.LayerNorm(config.embed_dim, epsilon=1e-6)\n",
        "        self.norm2 = nn.LayerNorm(config.embed_dim, epsilon=1e-6)\n",
        "        mlp_hidden_dim = int(config.embed_dim * config.mlp_ratio)\n",
        "\n",
        "        self.attn = create_attention_module(config, \"vision\")\n",
        "        self.mlp = VisionMlp(dim=config.embed_dim, hidden_dim=mlp_hidden_dim, hidden_act=config.hidden_act)\n",
        "\n",
        "    def forward(self, hidden_states, cu_seqlens, rotary_pos_emb) -> paddle.Tensor:\n",
        "        hidden_states = hidden_states + self.attn(\n",
        "            self.norm1(hidden_states), cu_seqlens=cu_seqlens, rotary_pos_emb=rotary_pos_emb\n",
        "        )\n",
        "        hidden_states = hidden_states + self.mlp(self.norm2(hidden_states))\n",
        "        return hidden_states\n",
        "\n",
        "\n",
        "# Copied from transformers.models.llama.modeling_llama._prepare_4d_causal_attention_mask_with_cache_position\n",
        "def _prepare_4d_causal_attention_mask_with_cache_position(\n",
        "    attention_mask: paddle.Tensor,\n",
        "    sequence_length: int,\n",
        "    target_length: int,\n",
        "    dtype: paddle.dtype,\n",
        "    min_dtype: float,\n",
        "    cache_position: paddle.Tensor,\n",
        "    batch_size: int,\n",
        "):\n",
        "    \"\"\"\n",
        "    Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape\n",
        "    `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing.\n",
        "\n",
        "    Args:\n",
        "        attention_mask (`paddle.Tensor`):\n",
        "            A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`.\n",
        "        sequence_length (`int`):\n",
        "            The sequence length being processed.\n",
        "        target_length (`int`):\n",
        "            The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet.\n",
        "        dtype (`paddle.dtype`):\n",
        "            The dtype to use for the 4D attention mask.\n",
        "        min_dtype (`float`):\n",
        "            The minimum value representable with the dtype `dtype`.\n",
        "        cache_position (`paddle.Tensor`):\n",
        "            Indices depicting the position of the input sequence tokens in the sequence.\n",
        "        batch_size (`paddle.Tensor`):\n",
        "            Batch size.\n",
        "    \"\"\"\n",
        "    if attention_mask is not None and attention_mask.dim() == 4:\n",
        "        # In this case we assume that the mask comes already in inverted form and requires no inversion or slicing.\n",
        "        causal_mask = attention_mask\n",
        "    else:\n",
        "        causal_mask = paddle.full([sequence_length, target_length], fill_value=min_dtype, dtype=dtype)\n",
        "        if sequence_length != 1:\n",
        "            causal_mask = paddle.triu(x=causal_mask, diagonal=1)\n",
        "        causal_mask *= paddle.arange(target_length) > cache_position.reshape([-1, 1])\n",
        "        causal_mask = causal_mask[None, None, :, :].expand(shape=[batch_size, 1, -1, -1])\n",
        "        if attention_mask is not None:\n",
        "            causal_mask = causal_mask.clone()\n",
        "            mask_length = tuple(attention_mask.shape)[-1]\n",
        "            padding_mask = causal_mask[:, :, :, :mask_length] + attention_mask[:, None, None, :]\n",
        "            padding_mask = padding_mask == 0\n",
        "            causal_mask[:, :, :, :mask_length] = causal_mask[:, :, :, :mask_length].masked_fill(\n",
        "                mask=padding_mask, value=min_dtype\n",
        "            )\n",
        "\n",
        "    return causal_mask"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZRV1KmLdsOe0"
      },
      "source": [
        "### 1.1.15 Qwen2RMSNorm\n",
        "\n",
        "**Qwen2RMSNorm**\n",
        "\n",
        "效于 T5LayerNorm，通过均方根归一化（RMSNorm）实现对输入的归一化：\n",
        "\n",
        "**实现思路：**\n",
        "\n",
        "对输入张量按最后一个维度计算均值平方差（方差）。\n",
        "\n",
        "通过平方差开平方得到归一化因子，避免数值不稳定。\n",
        "\n",
        "使用训练的权重参数调整归一化后的值。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4_AyiV3asvDY"
      },
      "outputs": [],
      "source": [
        "class Qwen2RMSNorm(nn.Layer):\n",
        "    def __init__(self, hidden_size, eps=1e-6):\n",
        "        \"\"\"\n",
        "        Qwen2RMSNorm is equivalent to T5LayerNorm\n",
        "        \"\"\"\n",
        "        super().__init__()\n",
        "        self.weight = paddle.create_parameter(\n",
        "            shape=[hidden_size],\n",
        "            dtype=paddle.get_default_dtype(),\n",
        "            default_initializer=nn.initializer.Constant(1.0),\n",
        "        )\n",
        "        self.variance_epsilon = eps\n",
        "\n",
        "    def forward(self, hidden_states):\n",
        "        if paddle.in_dynamic_mode():\n",
        "            with paddle.amp.auto_cast(False):\n",
        "                variance = hidden_states.astype(\"float32\").pow(2).mean(-1, keepdim=True)\n",
        "                hidden_states = paddle.rsqrt(variance + self.variance_epsilon) * hidden_states\n",
        "        else:\n",
        "            variance = hidden_states.astype(\"float32\").pow(2).mean(-1, keepdim=True)\n",
        "            hidden_states = paddle.rsqrt(variance + self.variance_epsilon) * hidden_states\n",
        "\n",
        "        if self.weight.dtype in [paddle.float16, paddle.bfloat16]:\n",
        "            hidden_states = paddle.cast(hidden_states, self.weight.dtype)\n",
        "        return hidden_states * self.weight"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pcUX7h-ks0no"
      },
      "source": [
        "### 1.1.16 Qwen2MLP\n",
        "\n",
        "**Qwen2MLP**\n",
        "\n",
        "实现基于门控激活的多层感知机（MLP）\n",
        "\n",
        "**结构：**\n",
        "输入通过两个线性层分别计算门控值（gate_proj）和变换值（up_proj）。\n",
        "\n",
        "激活函数（例如GELU或ReLU）应用于门控值。\n",
        "\n",
        "最终通过另一个线性层（down_proj）恢复到原始维度。\n",
        "\n",
        "**核心逻辑：**\n",
        "\n",
        "将非线性激活和线性投影相结合，增强了模型的表达能力。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1ezVwHB2s9Dd"
      },
      "outputs": [],
      "source": [
        "class Qwen2MLP(nn.Layer):\n",
        "    def __init__(self, config):\n",
        "        super().__init__()\n",
        "        self.hidden_size = config.hidden_size\n",
        "        self.intermediate_size = config.intermediate_size\n",
        "        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias_attr=False)\n",
        "        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias_attr=False)\n",
        "        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias_attr=False)\n",
        "        self.act_fn = ACT2FN[config.hidden_act]\n",
        "\n",
        "    def forward(self, hidden_state):\n",
        "        return self.down_proj(self.act_fn(self.gate_proj(hidden_state)) * self.up_proj(hidden_state))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mfw9YiDjtC_K"
      },
      "source": [
        "### 1.1.17 Qwen2VLAttention\n",
        "\n",
        "**Qwen2VLAttention**\n",
        "\n",
        "标准多头注意力机制，结合了旋转位置嵌入（RoPE）以增强长序列处理能力\n",
        "\n",
        "**主要模块：**\n",
        "投影层（q_proj，k_proj，v_proj）将输入特征投影为查询、键和值。\n",
        "\n",
        "使用旋转嵌入调整键和值以融入位置信息。\n",
        "\n",
        "点积注意力计算 softmax(QK^T / sqrt(d))。\n",
        "\n",
        "**支持功能：**\n",
        "\n",
        "缓存（past_key_value）：对长序列应用时复用计算结果。\n",
        "\n",
        "动态掩码（attention_mask）：处理因填充导致的无效区域。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TGaub-eFtWK8"
      },
      "outputs": [],
      "source": [
        "class Qwen2VLAttention(nn.Layer):\n",
        "    \"\"\"\n",
        "    Multi-headed attention from 'Attention Is All You Need' paper. Modified to use sliding window attention: Longformer\n",
        "    and \"Generating Long Sequences with Sparse Transformers\".\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, config: Qwen2VLConfig, layer_idx: Optional[int] = None):\n",
        "        super().__init__()\n",
        "        self.config = config\n",
        "        self.layer_idx = layer_idx\n",
        "        if layer_idx is None:\n",
        "            logger.warning_once(\n",
        "                f\"Instantiating {self.__class__.__name__} without passing `layer_idx` is not recommended and will \"\n",
        "                \"to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` \"\n",
        "                \"when creating this class.\"\n",
        "            )\n",
        "\n",
        "        self.hidden_size = config.hidden_size\n",
        "        self.num_heads = config.num_attention_heads\n",
        "        self.head_dim = self.hidden_size // self.num_heads\n",
        "        self.num_key_value_heads = config.num_key_value_heads\n",
        "        self.num_key_value_groups = self.num_heads // self.num_key_value_heads\n",
        "        self.max_position_embeddings = config.max_position_embeddings\n",
        "        self.rope_theta = config.rope_theta\n",
        "        self.is_causal = True\n",
        "        self.attention_dropout = config.attention_dropout\n",
        "        self.rope_scaling = config.rope_scaling\n",
        "\n",
        "        if (self.head_dim * self.num_heads) != self.hidden_size:\n",
        "            raise ValueError(\n",
        "                f\"hidden_size must be divisible by num_heads (got `hidden_size`: {self.hidden_size}\"\n",
        "                f\" and `num_heads`: {self.num_heads}).\"\n",
        "            )\n",
        "        self.q_proj = nn.Linear(self.hidden_size, self.num_heads * self.head_dim, bias_attr=True)\n",
        "        self.k_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias_attr=True)\n",
        "        self.v_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias_attr=True)\n",
        "        self.o_proj = nn.Linear(self.num_heads * self.head_dim, self.hidden_size, bias_attr=False)\n",
        "\n",
        "        self.rotary_emb = Qwen2RotaryEmbedding(\n",
        "            self.head_dim,\n",
        "            max_position_embeddings=self.max_position_embeddings,\n",
        "            base=self.rope_theta,\n",
        "        )\n",
        "\n",
        "    def forward(\n",
        "        self,\n",
        "        hidden_states: paddle.Tensor,\n",
        "        attention_mask: Optional[paddle.Tensor] = None,\n",
        "        position_ids: Optional[paddle.Tensor] = None,\n",
        "        past_key_value: Optional[Tuple[paddle.Tensor]] = None,  # Cache\n",
        "        output_attentions: bool = False,\n",
        "        use_cache: bool = False,  # default true\n",
        "        cache_position: Optional[paddle.Tensor] = None,\n",
        "    ) -> Tuple[paddle.Tensor, Optional[paddle.Tensor], Optional[Tuple[paddle.Tensor]]]:\n",
        "        bsz, q_len, _ = hidden_states.shape\n",
        "\n",
        "        try:\n",
        "            query_states = self.q_proj(hidden_states)\n",
        "            key_states = self.k_proj(hidden_states)\n",
        "            value_states = self.v_proj(hidden_states)\n",
        "        except:\n",
        "            hidden_states = hidden_states.astype(\"bfloat16\")\n",
        "            query_states = self.q_proj(hidden_states)\n",
        "            key_states = self.k_proj(hidden_states)\n",
        "            value_states = self.v_proj(hidden_states)\n",
        "\n",
        "        new_perm = [0, 2, 1, 3]\n",
        "        query_states = query_states.reshape([bsz, q_len, self.num_heads, self.head_dim]).transpose(new_perm)\n",
        "        key_states = key_states.reshape([bsz, q_len, self.num_key_value_heads, self.head_dim]).transpose(new_perm)\n",
        "        value_states = value_states.reshape([bsz, q_len, self.num_key_value_heads, self.head_dim]).transpose(new_perm)\n",
        "\n",
        "        kv_seq_len = key_states.shape[-2]  # q_len ######## [bs, num_head, seq_len, head_dim]      # qwen2是 [-3]\n",
        "        if past_key_value is not None:\n",
        "            kv_seq_len += cache_position[0] + 1\n",
        "            # kv_seq_len += past_key_value[0].shape[-2] # qwen2是 [-3]\n",
        "\n",
        "        cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)\n",
        "        query_states, key_states = apply_multimodal_rotary_pos_emb(\n",
        "            query_states, key_states, cos, sin, position_ids, self.rope_scaling[\"mrope_section\"]\n",
        "        )\n",
        "\n",
        "        # [bs, num_head, seq_len, head_dim]\n",
        "        if past_key_value is not None:\n",
        "            # cache_kwargs = {\"sin\": sin, \"cos\": cos, \"cache_position\": cache_position}  # Specific to RoPE models\n",
        "            # key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)\n",
        "            key_states = paddle.concat([past_key_value[0], key_states], axis=2)  # qwen2是 axis=1, qwen2_vl是 axis=2\n",
        "            value_states = paddle.concat([past_key_value[1], value_states], axis=2)  # qwen2是 axis=1\n",
        "        past_key_value = (key_states, value_states) if use_cache else None\n",
        "\n",
        "        # repeat k/v heads if n_kv_heads < n_heads\n",
        "        key_states = repeat_kv(key_states, self.num_key_value_groups)\n",
        "        value_states = repeat_kv(value_states, self.num_key_value_groups)\n",
        "\n",
        "        attn_weights = paddle.matmul(query_states, key_states.transpose([0, 1, 3, 2])) / math.sqrt(self.head_dim)\n",
        "\n",
        "        if attention_mask is not None:  # no matter the length, we just slice it\n",
        "            causal_mask = attention_mask[:, :, :, : key_states.shape[-2]]\n",
        "            attn_weights = attn_weights + causal_mask\n",
        "\n",
        "        # upcast attention to fp32\n",
        "        attn_weights = nn.functional.softmax(x=attn_weights, axis=-1, dtype=\"float32\")\n",
        "        attn_weights = nn.functional.dropout(x=attn_weights, p=self.attention_dropout, training=self.training)\n",
        "        attn_output = paddle.matmul(attn_weights.cast(\"bfloat16\"), value_states.cast(\"bfloat16\"))  # TODO: hard code\n",
        "\n",
        "        if attn_output.shape != [bsz, self.num_heads, q_len, self.head_dim]:\n",
        "            raise ValueError(\n",
        "                f\"`attn_output` should be of size {(bsz, q_len, self.num_heads, self.head_dim)}, but is\"\n",
        "                f\" {attn_output.shape}\"\n",
        "            )\n",
        "\n",
        "        attn_output = attn_output.transpose([0, 2, 1, 3])\n",
        "        attn_output = attn_output.reshape([bsz, q_len, -1])\n",
        "        attn_output = self.o_proj(attn_output)\n",
        "        if not output_attentions:\n",
        "            attn_weights = None\n",
        "        return attn_output, attn_weights, past_key_value"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DqgTLNbrtjdE"
      },
      "source": [
        "### 1.1.18 Qwen2VLFlashAttention2\n",
        "\n",
        "**Qwen2VLFlashAttention2**\n",
        "\n",
        "基于 **Flash Attention** 的实现，适用于更高效的序列建模"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ugHkWJl6tuQz"
      },
      "outputs": [],
      "source": [
        "class Qwen2VLFlashAttention2(Qwen2VLAttention):\n",
        "    \"\"\"\n",
        "    Qwen2VL flash attention module, following Qwen2VL attention module. This module inherits from `Qwen2VLAttention`\n",
        "    as the weights of the module stays untouched. The only required change would be on the forward pass\n",
        "    where it needs to correctly call the public API of flash attention and deal with padding tokens\n",
        "    in case the input contains any of them. Additionally, for sliding window attention, we apply SWA only to the bottom\n",
        "    config.max_window_layers layers.\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, *args, **kwargs):\n",
        "        super().__init__(*args, **kwargs)\n",
        "\n",
        "    def forward(\n",
        "        self,\n",
        "        hidden_states: paddle.Tensor,\n",
        "        attention_mask: Optional[paddle.Tensor] = None,\n",
        "        position_ids: Optional[paddle.Tensor] = None,\n",
        "        past_key_value: Optional[Tuple[paddle.Tensor]] = None,  # Cache\n",
        "        output_attentions: bool = False,\n",
        "        use_cache: bool = False,  # default true\n",
        "        cache_position: Optional[paddle.Tensor] = None,\n",
        "    ) -> Tuple[paddle.Tensor, Optional[paddle.Tensor], Optional[Tuple[paddle.Tensor]]]:\n",
        "        bsz, q_len, _ = tuple(hidden_states.shape)\n",
        "\n",
        "        try:\n",
        "            query_states = self.q_proj(hidden_states)\n",
        "            key_states = self.k_proj(hidden_states)\n",
        "            value_states = self.v_proj(hidden_states)\n",
        "        except:\n",
        "            hidden_states = hidden_states.astype(\"bfloat16\")\n",
        "            query_states = self.q_proj(hidden_states)\n",
        "            key_states = self.k_proj(hidden_states)\n",
        "            value_states = self.v_proj(hidden_states)\n",
        "\n",
        "        new_perm = [0, 2, 1, 3]\n",
        "        # [1, 3599, 1536] [bsz, q_len, self.num_heads * self.head_dim]\n",
        "        query_states = query_states.reshape([bsz, q_len, self.num_heads, self.head_dim]).transpose(new_perm)\n",
        "        key_states = key_states.reshape([bsz, q_len, self.num_key_value_heads, self.head_dim]).transpose(new_perm)\n",
        "        value_states = value_states.reshape([bsz, q_len, self.num_key_value_heads, self.head_dim]).transpose(new_perm)\n",
        "\n",
        "        kv_seq_len = key_states.shape[-2]  # q_len ######## [bs, num_head, seq_len, head_dim]      # qwen2是 [-3]\n",
        "        if past_key_value is not None:\n",
        "            kv_seq_len += cache_position[0] + 1\n",
        "\n",
        "        # Because the input can be padded, the absolute sequence length depends on the max position id.\n",
        "        cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)\n",
        "        query_states, key_states = apply_multimodal_rotary_pos_emb(\n",
        "            query_states, key_states, cos, sin, position_ids, self.rope_scaling[\"mrope_section\"]\n",
        "        )\n",
        "\n",
        "        if past_key_value is not None:\n",
        "            # cache_kwargs = {\"sin\": sin, \"cos\": cos, \"cache_position\": cache_position}  # Specific to RoPE models\n",
        "            # key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)\n",
        "            key_states = paddle.concat([past_key_value[0], key_states], axis=2)  # qwen2是 axis=1, qwen2_vl是 axis=2\n",
        "            value_states = paddle.concat([past_key_value[1], value_states], axis=2)  # qwen2是 axis=1\n",
        "        past_key_value = (key_states, value_states) if use_cache else None\n",
        "\n",
        "        # repeat k/v heads if n_kv_heads < n_heads\n",
        "        key_states = repeat_kv(key_states, self.num_key_value_groups)\n",
        "        value_states = repeat_kv(value_states, self.num_key_value_groups)\n",
        "\n",
        "        # Reashape to the expected shape for Flash Attention\n",
        "        # [1, 3599, 12, 128]\n",
        "        query_states = query_states.transpose(perm=[0, 2, 1, 3])\n",
        "        key_states = key_states.transpose(perm=[0, 2, 1, 3])\n",
        "        value_states = value_states.transpose(perm=[0, 2, 1, 3])\n",
        "\n",
        "        attn_output = self._flash_attention_forward(\n",
        "            query_states,\n",
        "            key_states,\n",
        "            value_states,\n",
        "            None,\n",
        "            q_len\n",
        "            # dropout=0.0 if not self.training else self.attention_dropout,\n",
        "            # causal=self.is_causal,\n",
        "        )\n",
        "\n",
        "        attn_output = attn_output.reshape([bsz, q_len, self.hidden_size])\n",
        "        attn_output = self.o_proj(attn_output)\n",
        "        if not output_attentions:\n",
        "            attn_weights = None\n",
        "        return attn_output, attn_weights, past_key_value\n",
        "\n",
        "    def _flash_attention_forward(\n",
        "        self, query_states, key_states, value_states, attention_mask, query_length, dropout=0.0, softmax_scale=None\n",
        "    ):\n",
        "        \"\"\"\n",
        "        Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token\n",
        "        first unpad the input, then computes the attention scores and pad the final attention scores.\n",
        "\n",
        "        Args:\n",
        "            query_states (`paddle.Tensor`):\n",
        "                Input query states to be passed to Flash Attention API\n",
        "            key_states (`paddle.Tensor`):\n",
        "                Input key states to be passed to Flash Attention API\n",
        "            value_states (`paddle.Tensor`):\n",
        "                Input value states to be passed to Flash Attention API\n",
        "            attention_mask (`paddle.Tensor`):\n",
        "                The padding mask - corresponds to a tensor of size `(batch_size, seq_len)` where 0 stands for the\n",
        "                position of padding tokens and 1 for the position of non-padding tokens.\n",
        "            dropout (`int`, *optional*):\n",
        "                Attention dropout\n",
        "            softmax_scale (`float`, *optional*):\n",
        "                The scaling of QK^T before applying softmax. Default to 1 / sqrt(head_dim)\n",
        "        \"\"\"\n",
        "        # Contains at least one padding token in the sequence\n",
        "        causal = self.is_causal and query_length != 1\n",
        "\n",
        "        head_dim = query_states.shape[-1]\n",
        "        softmax_scale = head_dim**-0.5  # TODO: 需要手动加上\n",
        "\n",
        "        if attention_mask is not None:\n",
        "            batch_size = query_states.shape[0]  # [2, 3383, 16, 128]\n",
        "\n",
        "            query_states, key_states, value_states, indices_q, cu_seq_lens, max_seq_lens = self._unpad_input(\n",
        "                query_states, key_states, value_states, attention_mask, query_length\n",
        "            )\n",
        "\n",
        "            cu_seqlens_q, cu_seqlens_k = cu_seq_lens\n",
        "            max_seqlen_in_batch_q, max_seqlen_in_batch_k = max_seq_lens\n",
        "\n",
        "            attn_output_unpad = flash_attn_varlen_func(  # TODO: flash_attn_unpadded\n",
        "                query_states,  # [5998, 16, 128]\n",
        "                key_states,  # [5998, 8, 128]\n",
        "                value_states,  # [5998, 8, 128]\n",
        "                cu_seqlens_q=cu_seqlens_q,\n",
        "                cu_seqlens_k=cu_seqlens_k,\n",
        "                max_seqlen_q=max_seqlen_in_batch_q,\n",
        "                max_seqlen_k=max_seqlen_in_batch_k,\n",
        "                scale=softmax_scale,  # not softmax_scale=\n",
        "                dropout=dropout,\n",
        "                causal=causal,\n",
        "            )[0]\n",
        "\n",
        "            attn_output = pad_input(attn_output_unpad, indices_q, batch_size, query_length)\n",
        "        else:\n",
        "            attn_output = flash_attn_func(\n",
        "                query_states,\n",
        "                key_states,\n",
        "                value_states,\n",
        "                dropout,\n",
        "                causal=causal,  # no softmax_scale=\n",
        "            )[0]\n",
        "\n",
        "        return attn_output\n",
        "\n",
        "    def _unpad_input(self, query_layer, key_layer, value_layer, attention_mask, query_length):\n",
        "        indices_k, cu_seqlens_k, max_seqlen_in_batch_k = _get_unpad_data(attention_mask)\n",
        "        batch_size, kv_seq_len, num_key_value_heads, head_dim = key_layer.shape\n",
        "\n",
        "        key_layer = index_first_axis(\n",
        "            key_layer.reshape([batch_size * kv_seq_len, num_key_value_heads, head_dim]), indices_k\n",
        "        )\n",
        "        value_layer = index_first_axis(\n",
        "            value_layer.reshape([batch_size * kv_seq_len, num_key_value_heads, head_dim]), indices_k\n",
        "        )\n",
        "\n",
        "        if query_length == kv_seq_len:\n",
        "            query_layer = index_first_axis(\n",
        "                query_layer.reshape([batch_size * kv_seq_len, self.num_heads, head_dim]), indices_k\n",
        "            )\n",
        "            cu_seqlens_q = cu_seqlens_k\n",
        "            max_seqlen_in_batch_q = max_seqlen_in_batch_k\n",
        "            indices_q = indices_k\n",
        "        elif query_length == 1:\n",
        "            max_seqlen_in_batch_q = 1\n",
        "            cu_seqlens_q = paddle.arange(\n",
        "                batch_size + 1, dtype=paddle.int32\n",
        "            )  # There is a memcpy here, that is very bad.\n",
        "            indices_q = cu_seqlens_q[:-1]\n",
        "            query_layer = query_layer.squeeze(1)\n",
        "        else:\n",
        "            # The -q_len: slice assumes left padding.\n",
        "            attention_mask = attention_mask[:, -query_length:]\n",
        "            query_layer, indices_q, cu_seqlens_q, max_seqlen_in_batch_q = unpad_input(query_layer, attention_mask)\n",
        "\n",
        "        return (\n",
        "            query_layer,\n",
        "            key_layer,\n",
        "            value_layer,\n",
        "            indices_q.to(paddle.int64),\n",
        "            (cu_seqlens_q, cu_seqlens_k),\n",
        "            (max_seqlen_in_batch_q, max_seqlen_in_batch_k),\n",
        "        )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VIMc7ZyVuPFz"
      },
      "source": [
        "### 1.1.19 Qwen2VLDecoderLayer\n",
        "\n",
        "**Qwen2VLDecoderLayer**\n",
        "\n",
        "**功能**\n",
        "\n",
        "这是一个Transformer解码器的实现，包含两个主要部分：\n",
        "\n",
        "**自注意力模块（Self Attention）：**用于捕获输入序列中各位置之间的全局依赖关系。\n",
        "\n",
        "**前馈神经网络模块（Feed-Forward Network, FFN）：**用于非线性变换，增强表示能力。\n",
        "\n",
        "\n",
        "**self.self_attn:** 自注意力机制模块。\n",
        "\n",
        "**self.mlp:** 两层线性变换和激活函数构成的前馈神经网络。\n",
        "\n",
        "**LayerNorm:** 使用RMS归一化方法对输入数据进行标准化。\n",
        "\n",
        "**残差连接:** 将每个子层的输入加到输出中，以缓解梯度消失问题并促进信息流动。\n",
        "\n",
        "**前向传播（forward）流程**\n",
        "\n",
        "1.对输入状态进行标准化（self.input_layernorm）。\n",
        "\n",
        "2.通过自注意力模块，计算更新后的隐藏状态。\n",
        "\n",
        "3.对自注意力输出进行标准化，再通过前馈网络进一步处理。\n",
        "\n",
        "4.将最终的隐藏状态与残差相加，得到模块输出。\n",
        "\n",
        "5.返回隐藏状态以及（可选的）注意力权重和缓存。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EGvjEVS0ushj"
      },
      "outputs": [],
      "source": [
        "class Qwen2VLDecoderLayer(nn.Layer):\n",
        "    def __init__(self, config: Qwen2VLConfig, layer_idx: int):\n",
        "        super().__init__()\n",
        "        self.hidden_size = config.hidden_size\n",
        "\n",
        "        # use_sliding_window false\n",
        "        if config.use_sliding_window and config.attn_implementation != \"flash_attention_2\":\n",
        "            logger.warning_once(\n",
        "                f\"Sliding Window Attention is enabled but not implemented for `{config.attn_implementation}`; \"\n",
        "                \"unexpected results may be encountered.\"\n",
        "            )\n",
        "        self.self_attn = create_attention_module(config, \"qwen2vl\", layer_idx=layer_idx)\n",
        "        self.mlp = Qwen2MLP(config)\n",
        "        self.input_layernorm = Qwen2RMSNorm(config.hidden_size, eps=config.rms_norm_eps)\n",
        "        self.post_attention_layernorm = Qwen2RMSNorm(config.hidden_size, eps=config.rms_norm_eps)\n",
        "\n",
        "    def forward(\n",
        "        self,\n",
        "        hidden_states: paddle.Tensor,\n",
        "        attention_mask: Optional[paddle.Tensor] = None,\n",
        "        position_ids: Optional[paddle.Tensor] = None,\n",
        "        past_key_value: Optional[Tuple[paddle.Tensor]] = None,\n",
        "        output_attentions: Optional[bool] = False,\n",
        "        use_cache: Optional[bool] = False,\n",
        "        cache_position: Optional[paddle.Tensor] = None,\n",
        "        **kwargs,\n",
        "    ):\n",
        "        \"\"\"\n",
        "        Args:\n",
        "            hidden_states (`paddle.Tensor`): input to the layer of shape `(batch, seq_len, embed_dim)`\n",
        "            attention_mask (`paddle.Tensor`, *optional*): attention mask of size\n",
        "                `(batch, sequence_length)` where padding elements are indicated by 0.\n",
        "            output_attentions (`bool`, *optional*):\n",
        "                Whether or not to return the attentions tensors of all attention layers. See `attentions` under\n",
        "                returned tensors for more detail.\n",
        "            use_cache (`bool`, *optional*):\n",
        "                If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding\n",
        "                (see `past_key_values`).\n",
        "            past_key_value (`Tuple(paddle.Tensor)`, *optional*): cached past key and value projection states\n",
        "            cache_position (`paddle.Tensor` of shape `(sequence_length)`, *optional*):\n",
        "                Indices depicting the position of the input sequence tokens in the sequence.\n",
        "            kwargs (`dict`, *optional*):\n",
        "                Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code\n",
        "                into the model\n",
        "        \"\"\"\n",
        "\n",
        "        residual = hidden_states\n",
        "\n",
        "        hidden_states = self.input_layernorm(hidden_states)\n",
        "\n",
        "        # Self Attention\n",
        "        hidden_states, self_attn_weights, present_key_value = self.self_attn(\n",
        "            hidden_states=hidden_states,\n",
        "            attention_mask=attention_mask,\n",
        "            position_ids=position_ids,\n",
        "            past_key_value=past_key_value,\n",
        "            output_attentions=output_attentions,\n",
        "            use_cache=use_cache,\n",
        "            cache_position=cache_position,\n",
        "        )\n",
        "        hidden_states = residual + hidden_states\n",
        "\n",
        "        # Fully Connected\n",
        "        residual = hidden_states\n",
        "        hidden_states = self.post_attention_layernorm(hidden_states)\n",
        "        hidden_states = self.mlp(hidden_states)\n",
        "        hidden_states = residual + hidden_states\n",
        "\n",
        "        outputs = (hidden_states,)\n",
        "\n",
        "        if output_attentions:\n",
        "            outputs += (self_attn_weights,)\n",
        "\n",
        "        if use_cache:\n",
        "            outputs += (present_key_value,)\n",
        "\n",
        "        return outputs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VksqjJvPu2OH"
      },
      "source": [
        "### 1.1.20 Qwen2VisionTransformerPretrainedModel\n",
        "\n",
        "**Qwen2VisionTransformerPretrainedModel**\n",
        "\n",
        "**功能**\n",
        "实现了一个视觉Transformer，用于处理图像数据并提取视觉特征。这部分与Transformer编码器类似，但适配了视觉任务需求。\n",
        "\n",
        "**组成**\n",
        "\n",
        "**Patch Embedding (self.patch_embed):** 将输入图像切分为小块（patches）并映射到高维嵌入空间。\n",
        "\n",
        "**Rotary Positional Embedding (self.rotary_pos_emb):** 提供旋转式的位置编码，用于捕捉输入序列的位置信息。\n",
        "\n",
        "**Blocks (self.blocks):** 多层堆叠的视觉Transformer块，完成对图像特征的逐步提取和处理。\n",
        "\n",
        "**Merger (self.merger):** 合并多尺度的特征信息。\n",
        "\n",
        "**前向传播（forward）流程**\n",
        "\n",
        "**嵌入生成:** 使用PatchEmbed将输入图像转换为高维特征表示。\n",
        "\n",
        "**位置编码:** 计算输入序列的旋转式位置嵌入。\n",
        "\n",
        "**逐层处理:** 依次通过视觉Transformer块，更新隐藏状态。\n",
        "\n",
        "**特征合并:** 使用PatchMerger模块将提取的多尺度视觉特征整合成最终表示。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "F3AzZwJgvsIB"
      },
      "outputs": [],
      "source": [
        "class Qwen2VisionTransformerPretrainedModel(Qwen2VLPreTrainedModel):\n",
        "    config_class = Qwen2VLVisionConfig\n",
        "    _no_split_modules = [\"Qwen2VLVisionBlock\"]\n",
        "\n",
        "    def __init__(self, config) -> None:\n",
        "        super().__init__(config)\n",
        "        self.spatial_merge_size = config.spatial_merge_size\n",
        "\n",
        "        self.patch_embed = PatchEmbed(\n",
        "            patch_size=config.patch_size,\n",
        "            temporal_patch_size=config.temporal_patch_size,\n",
        "            in_channels=config.in_channels,\n",
        "            embed_dim=config.embed_dim,\n",
        "        )\n",
        "\n",
        "        head_dim = config.embed_dim // config.num_heads\n",
        "        self.rotary_pos_emb = VisionRotaryEmbedding(head_dim // 2)\n",
        "\n",
        "        self.blocks = nn.LayerList([Qwen2VLVisionBlock(config) for _ in range(config.depth)])\n",
        "        self.merger = PatchMerger(dim=config.hidden_size, context_dim=config.embed_dim)\n",
        "\n",
        "    def get_dtype(self) -> paddle.dtype:\n",
        "        return self.blocks[0].mlp.fc2.weight.dtype\n",
        "\n",
        "    def rot_pos_emb(self, grid_thw):\n",
        "        pos_ids = []\n",
        "        for t, h, w in grid_thw:\n",
        "            hpos_ids = paddle.arange(h).unsqueeze(1).expand([-1, w])\n",
        "            hpos_ids = hpos_ids.reshape(\n",
        "                [\n",
        "                    h // self.spatial_merge_size,\n",
        "                    self.spatial_merge_size,\n",
        "                    w // self.spatial_merge_size,\n",
        "                    self.spatial_merge_size,\n",
        "                ]\n",
        "            )\n",
        "            hpos_ids = hpos_ids.transpose(perm=[0, 2, 1, 3])\n",
        "            hpos_ids = hpos_ids.flatten()\n",
        "\n",
        "            wpos_ids = paddle.arange(w).unsqueeze(0).expand([h, -1])\n",
        "            wpos_ids = wpos_ids.reshape(\n",
        "                [\n",
        "                    h // self.spatial_merge_size,\n",
        "                    self.spatial_merge_size,\n",
        "                    w // self.spatial_merge_size,\n",
        "                    self.spatial_merge_size,\n",
        "                ]\n",
        "            )\n",
        "            wpos_ids = wpos_ids.transpose([0, 2, 1, 3])\n",
        "            wpos_ids = wpos_ids.flatten()\n",
        "            pos_ids.append(paddle.stack(x=[hpos_ids, wpos_ids], axis=-1).tile(repeat_times=[t, 1]))\n",
        "        pos_ids = paddle.concat(x=pos_ids, axis=0)\n",
        "        max_grid_size = grid_thw[:, 1:].max()\n",
        "        rotary_pos_emb_full = self.rotary_pos_emb(max_grid_size)\n",
        "        rotary_pos_emb = rotary_pos_emb_full[pos_ids].flatten(start_axis=1)\n",
        "        return rotary_pos_emb\n",
        "\n",
        "    def forward(self, hidden_states: paddle.Tensor, grid_thw: paddle.Tensor) -> paddle.Tensor:\n",
        "\n",
        "        hidden_states = self.patch_embed(hidden_states)\n",
        "        rotary_pos_emb = self.rot_pos_emb(grid_thw)\n",
        "\n",
        "        cu_seqlens = paddle.repeat_interleave(grid_thw[:, 1] * grid_thw[:, 2], grid_thw[:, 0]).cumsum(\n",
        "            axis=0, dtype=\"int32\"\n",
        "        )\n",
        "        cu_seqlens = F.pad(cu_seqlens, (1, 0), value=0)\n",
        "\n",
        "        for idx, blk in enumerate(self.blocks):\n",
        "            hidden_states = blk(hidden_states, cu_seqlens=cu_seqlens, rotary_pos_emb=rotary_pos_emb)\n",
        "\n",
        "        return self.merger(hidden_states)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bK594fBbv-gt"
      },
      "source": [
        "### 1.1.21 Qwen2VLModel\n",
        "\n",
        "**Qwen2VLModel**\n",
        "\n",
        "该类是一个自定义的解码器模型，主要用于处理序列生成任务，如语言模型或序列到序列模型。\n",
        "\n",
        "关键属性与方法\n",
        "\n",
        "__init__ 构造方法:\n",
        "\n",
        "**初始化模型，**包括词嵌入层（self.embed_tokens）、多层解码器层（self.layers）、归一化层（self.norm）以及其他配置参数。\n",
        "\n",
        "**padding_idx:** 用于处理序列中填充位置（如 PAD token）的索引。\n",
        "\n",
        "**gradient_checkpointing:** 控制是否启用梯度检查点技术，以减少显存占用。\n",
        "\n",
        "**get_input_embeddings 和 set_input_embeddings:**获取或设置输入的嵌入层，用于与其他模块共享词嵌入权重。\n",
        "\n",
        "**_prepare_decoder_attention_mask:**\n",
        "准备解码器使用的注意力掩码，支持两种掩码\n",
        "\n",
        "**扩展的输入注意力掩码:** 用于屏蔽填充位置。\n",
        "\n",
        "**因果掩码:** 确保解码器在生成序列时只能访问过去的信息（自回归特性）。\n",
        "\n",
        "\n",
        "**forward 方法:**\n",
        "\n",
        "核心计算逻辑，完成模型的前向传播。主要步骤包括：\n",
        "\n",
        "**输入处理:** 接受 input_ids 或嵌入表示 inputs_embeds。\n",
        "\n",
        "**嵌入生成:** 将输入序列映射为嵌入表示。\n",
        "\n",
        "**注意力掩码计算:** 根据输入生成因果掩码和填充掩码。\n",
        "\n",
        "**位置编码:** 通过 position_ids 提供位置信息。\n",
        "\n",
        "**解码层堆叠:** 逐层处理隐藏状态，生成解码器输出。\n",
        "\n",
        "**返回结果:** 支持多种返回格式，包括隐层状态、注意力权重和缓存。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wz_-PlAMvz-P"
      },
      "outputs": [],
      "source": [
        "class Qwen2VLModel(Qwen2VLPreTrainedModel):\n",
        "    def __init__(self, config: Qwen2VLConfig):\n",
        "        super().__init__(config)\n",
        "        self.padding_idx = config.pad_token_id\n",
        "        self.vocab_size = config.vocab_size\n",
        "\n",
        "        self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)\n",
        "        self.layers = nn.LayerList(\n",
        "            [Qwen2VLDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]\n",
        "        )\n",
        "        self.norm = Qwen2RMSNorm(config.hidden_size, eps=config.rms_norm_eps)\n",
        "\n",
        "        self.gradient_checkpointing = False\n",
        "\n",
        "    def get_input_embeddings(self):\n",
        "        return self.embed_tokens\n",
        "\n",
        "    def set_input_embeddings(self, value):\n",
        "        self.embed_tokens = value\n",
        "\n",
        "    @staticmethod\n",
        "    def _prepare_decoder_attention_mask(attention_mask, input_shape, past_key_values_length, dtype):\n",
        "        if attention_mask is not None:\n",
        "            # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]\n",
        "            if len(attention_mask.shape) == 2:\n",
        "                expanded_attn_mask = _expand_2d_mask(attention_mask, dtype, tgt_length=input_shape[-1])\n",
        "                # For decoding phase in generation, seq_length = 1, we don't need to add causal mask\n",
        "                if input_shape[-1] > 1:\n",
        "                    combined_attention_mask = _make_causal_mask(\n",
        "                        input_shape,\n",
        "                        past_key_values_length=past_key_values_length,\n",
        "                    )\n",
        "                    expanded_attn_mask = expanded_attn_mask & combined_attention_mask\n",
        "            # [bsz, seq_len, seq_len] -> [bsz, 1, seq_len, seq_len]\n",
        "            elif len(attention_mask.shape) == 3:\n",
        "                expanded_attn_mask = attention_mask.unsqueeze(1).astype(\"bool\")\n",
        "            # if attention_mask is already 4-D, do nothing\n",
        "            else:\n",
        "                expanded_attn_mask = attention_mask\n",
        "        else:\n",
        "            expanded_attn_mask = _make_causal_mask(\n",
        "                input_shape,\n",
        "                past_key_values_length=past_key_values_length,\n",
        "            )\n",
        "        # Convert bool attention_mask to float attention mask, which will be added to attention_scores later\n",
        "        expanded_attn_mask = paddle.where(expanded_attn_mask, 0.0, paddle.finfo(dtype).min).astype(dtype)\n",
        "        return expanded_attn_mask\n",
        "\n",
        "    def forward(\n",
        "        self,\n",
        "        input_ids: paddle.Tensor = None,\n",
        "        attention_mask: Optional[paddle.Tensor] = None,\n",
        "        position_ids: Optional[paddle.Tensor] = None,\n",
        "        past_key_values: Optional[List[paddle.Tensor]] = None,\n",
        "        inputs_embeds: Optional[paddle.Tensor] = None,\n",
        "        use_cache: Optional[bool] = None,\n",
        "        output_attentions: Optional[bool] = None,\n",
        "        output_hidden_states: Optional[bool] = None,\n",
        "        return_dict: Optional[bool] = None,\n",
        "        cache_position: Optional[paddle.Tensor] = None,\n",
        "    ) -> Union[Tuple, BaseModelOutputWithPast]:\n",
        "        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions\n",
        "        output_hidden_states = (\n",
        "            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states\n",
        "        )\n",
        "        use_cache = use_cache if use_cache is not None else self.config.use_cache\n",
        "\n",
        "        return_dict = return_dict if return_dict is not None else self.config.use_return_dict\n",
        "\n",
        "        if (input_ids is None) ^ (inputs_embeds is not None):\n",
        "            raise ValueError(\n",
        "                \"You cannot specify both input_ids and inputs_embeds at the same time, and must specify either one\"\n",
        "            )\n",
        "        elif input_ids is not None:\n",
        "            batch_size, seq_length = input_ids.shape\n",
        "        elif inputs_embeds is not None:\n",
        "            batch_size, seq_length, _ = inputs_embeds.shape\n",
        "        else:\n",
        "            raise ValueError(\"You have to specify either decoder_input_ids or decoder_inputs_embeds\")\n",
        "\n",
        "        if past_key_values is None:\n",
        "            past_key_values = tuple([None] * len(self.layers))\n",
        "        # NOTE: to make cache can be clear in-time\n",
        "        past_key_values = list(past_key_values)\n",
        "\n",
        "        seq_length_with_past = seq_length\n",
        "        cache_length = 0\n",
        "        if past_key_values[0] is not None:\n",
        "            cache_length = past_key_values[0][0].shape[2]  # shape[1] in qwen2\n",
        "            seq_length_with_past += cache_length\n",
        "\n",
        "        if inputs_embeds is None:\n",
        "            inputs_embeds = self.embed_tokens(input_ids)\n",
        "\n",
        "        # embed positions\n",
        "        if attention_mask is None:\n",
        "            # [bs, seq_len]\n",
        "            attention_mask = paddle.ones((batch_size, seq_length_with_past), dtype=paddle.bool)\n",
        "\n",
        "        attention_mask = self._prepare_decoder_attention_mask(\n",
        "            attention_mask, (batch_size, seq_length), cache_length, inputs_embeds.dtype\n",
        "        )  # [bs, 1, seq_len, seq_len]\n",
        "        causal_mask = attention_mask\n",
        "\n",
        "        if cache_position is None:\n",
        "            past_seen_tokens = past_key_values[0][0].shape[2] if past_key_values[0] is not None else 0\n",
        "            cache_position = paddle.arange(past_seen_tokens, past_seen_tokens + inputs_embeds.shape[1])\n",
        "\n",
        "        if position_ids is None:\n",
        "            # the hard coded `3` is for temporal, height and width.\n",
        "            position_ids = cache_position.reshape([1, 1, -1]).expand([3, inputs_embeds.shape[0], -1])\n",
        "\n",
        "        hidden_states = inputs_embeds\n",
        "\n",
        "        # decoder layers\n",
        "        all_hidden_states = () if output_hidden_states else None\n",
        "        all_self_attns = () if output_attentions else None\n",
        "        next_decoder_cache = ()\n",
        "\n",
        "        for idx, (decoder_layer) in enumerate(self.layers):\n",
        "            if output_hidden_states:\n",
        "                all_hidden_states += (hidden_states,)\n",
        "\n",
        "            past_key_value = past_key_values[idx] if past_key_values is not None else None\n",
        "\n",
        "            layer_outputs = decoder_layer(\n",
        "                hidden_states,\n",
        "                attention_mask=causal_mask,\n",
        "                position_ids=position_ids,\n",
        "                past_key_value=past_key_value,\n",
        "                output_attentions=output_attentions,  # False\n",
        "                use_cache=use_cache,  # True\n",
        "                cache_position=cache_position,\n",
        "            )\n",
        "\n",
        "            # NOTE: clear outdate cache after it has been used for memory saving\n",
        "            past_key_value = past_key_values[idx] = None\n",
        "\n",
        "            hidden_states = layer_outputs[0]\n",
        "\n",
        "            next_decoder_cache = next_decoder_cache + (layer_outputs[-1],) if use_cache else None\n",
        "\n",
        "            if output_attentions:\n",
        "                all_self_attns += (layer_outputs[1],)\n",
        "\n",
        "        hidden_states = self.norm(hidden_states)\n",
        "\n",
        "        # add hidden states from the last decoder layer\n",
        "        if output_hidden_states:\n",
        "            all_hidden_states += (hidden_states,)\n",
        "\n",
        "        next_cache = next_decoder_cache if use_cache else None\n",
        "\n",
        "        if not return_dict:\n",
        "            return tuple(v for v in [hidden_states, next_cache, all_hidden_states, all_self_attns] if v is not None)\n",
        "        return BaseModelOutputWithPast(\n",
        "            last_hidden_state=hidden_states,\n",
        "            past_key_values=next_cache,\n",
        "            hidden_states=all_hidden_states,\n",
        "            attentions=all_self_attns,\n",
        "        )\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h17VR74pxgGa"
      },
      "source": [
        "### 1.1.22 Qwen2LMHead\n",
        "\n",
        "**Qwen2LMHead**\n",
        "\n",
        "该类实现了一个语言建模头，用于将解码器的隐藏状态转换为词汇表上的分布（即输出概率）\n",
        "\n",
        "**关键属性与方法**\n",
        "\n",
        "__init__ 构造方法:\n",
        "\n",
        "\n",
        "初始化模型的权重矩阵 self.weight。\n",
        "\n",
        "**提供两种权重存储形式：**\n",
        "\n",
        "转置形式 (transpose_y=True): 通常用于共享输入嵌入层的权重。\n",
        "\n",
        "标准形式 (transpose_y=False): 用于独立定义输出层的权重。\n",
        "\n",
        "**forward 方法:**\n",
        "\n",
        "通过矩阵乘法，将解码器的隐藏状态映射到词汇表分布。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ap84vYzwxs2G"
      },
      "outputs": [],
      "source": [
        "class Qwen2LMHead(nn.Layer):\n",
        "    def __init__(self, config, embedding_weights=None, transpose_y=False):\n",
        "        super(Qwen2LMHead, self).__init__()\n",
        "        self.config = config\n",
        "        vocab_size = config.vocab_size\n",
        "\n",
        "        self.transpose_y = transpose_y\n",
        "        if transpose_y:\n",
        "            # only for weight from embedding_weights\n",
        "            if embedding_weights is not None:\n",
        "                self.weight = embedding_weights\n",
        "            else:\n",
        "                self.weight = self.create_parameter(\n",
        "                    shape=[vocab_size, config.hidden_size],\n",
        "                    dtype=paddle.get_default_dtype(),\n",
        "                )\n",
        "        else:\n",
        "            # for weight from model init\n",
        "            self.weight = self.create_parameter(\n",
        "                shape=[config.hidden_size, vocab_size],\n",
        "                dtype=paddle.get_default_dtype(),\n",
        "            )\n",
        "\n",
        "    def forward(self, hidden_states):\n",
        "        logits = paddle.matmul(hidden_states, self.weight, transpose_y=self.transpose_y)\n",
        "        return logits"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bi4esVLoWfrr"
      },
      "source": [
        "## **1.2 视觉编码器**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jkg8tfn2WrLl"
      },
      "source": [
        "视觉编码器的作用是从图像中提取有意义的特征，并将其转换为适合语言模型的格式。Qwen2-VL 使用了视觉Transformer架构来处理图像数据。\n",
        "\n",
        "\n",
        "**视觉编码器配置**:**Qwen2VLVisionConfig**\n",
        "\n",
        "首先,**Qwen2VLVisionConfig** 类定义了视觉编码器的配置，它继承自 **PretrainedConfig** 类，允许加载预训练的模型配置。以下是该类的核心参数："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rNmXNlEAWzzZ"
      },
      "outputs": [],
      "source": [
        "class Qwen2VLVisionConfig(PretrainedConfig):\n",
        "    model_type = \"qwen2_vl\"\n",
        "\n",
        "    def __init__(self, depth=32, embed_dim=1280, hidden_size=3584, hidden_act=\"quick_gelu\", mlp_ratio=4, num_heads=16, in_channels=3, patch_size=14, spatial_merge_size=2, temporal_patch_size=2, attn_implementation=\"eager\", **kwargs):\n",
        "        super().__init__(**kwargs)\n",
        "        self.depth = depth\n",
        "        self.embed_dim = embed_dim\n",
        "        self.hidden_size = hidden_size\n",
        "        self.hidden_act = hidden_act\n",
        "        self.mlp_ratio = mlp_ratio\n",
        "        self.num_heads = num_heads\n",
        "        self.in_channels = in_channels\n",
        "        self.patch_size = patch_size\n",
        "        self.spatial_merge_size = spatial_merge_size\n",
        "        self.temporal_patch_size = temporal_patch_size\n",
        "        self.attn_implementation = attn_implementation\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PL2WC-rXW4u3"
      },
      "source": [
        "**关键参数解析:**\n",
        "\n",
        "**patch_size：**表示图像切割成小块的大小。Qwen2-VL 采用了视觉 Transformer（ViT）架构，因此将图像分为多个固定大小的 patch，每个 patch 会被映射为一个向量。\n",
        "\n",
        "**depth：**Transformer 的层数，决定了视觉编码器的深度。\n",
        "\n",
        "**embed_dim：**每个图像 patch 的嵌入维度。\n",
        "\n",
        "**hidden_size：**隐藏层的大小，决定了模型处理信息的能力。\n",
        "\n",
        "**num_heads：**注意力头的数量，影响 Transformer 注意力机制的并行性。\n",
        "\n",
        "**attn_implementation：**表示使用的注意力实现方式，eager 或 flash，前者为传统的计算方式，后者使用 flash attention 优化计算速度。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AlniqygvUpRt"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WW3kVU0ZUsj9"
      },
      "source": [
        "**视觉编码器的实现：**\n",
        "\n",
        "\n",
        "视觉编码器的实现包含了图像的切分、嵌入生成和位置编码，具体代码如下："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pRTwLSpkWly7"
      },
      "outputs": [],
      "source": [
        "class Qwen2VisionTransformerPretrainedModel(PreTrainedModel):\n",
        "    def __init__(self, config: Qwen2VLVisionConfig):\n",
        "        super().__init__(config)\n",
        "        self.config = config\n",
        "\n",
        "        # 图像分块并映射到嵌入\n",
        "        self.patch_embed = PatchEmbed(config.in_channels, config.embed_dim, config.patch_size)\n",
        "\n",
        "        # 视觉位置编码\n",
        "        self.position_embeddings = VisionRotaryEmbedding(config.embed_dim, config.hidden_size)\n",
        "\n",
        "        # Transformer Block\n",
        "        self.blocks = nn.ModuleList([Qwen2VLVisionBlock(config) for _ in range(config.depth)])\n",
        "\n",
        "        # 生成图像的最终嵌入表示\n",
        "        self.layernorm = nn.LayerNorm(config.hidden_size)\n",
        "        self.pooler = nn.AdaptiveAvgPool1d(1)\n",
        "\n",
        "    def forward(self, x):\n",
        "        # 生成图像的patch嵌入\n",
        "        x = self.patch_embed(x)\n",
        "        x = self.position_embeddings(x)\n",
        "\n",
        "        # 通过Transformer层\n",
        "        for block in self.blocks:\n",
        "            x = block(x)\n",
        "\n",
        "        # 最后的池化和层归一化\n",
        "        x = self.layernorm(x)\n",
        "        x = self.pooler(x)\n",
        "\n",
        "        return x\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k4hsxGclZoiz"
      },
      "source": [
        "**代码解析:**\n",
        "\n",
        "**PatchEmbed：**负责将图像分成多个 **patch** 并将每个**patch**转化为一个向量。这样就将二维图像转换为一维序列，类似于文本中的 token。\n",
        "\n",
        "**VisionRotaryEmbedding：**这部分是位置编码模块，用于为每个**patch**添加位置信息，使模型能够处理不同位置的图像。\n",
        "\n",
        "**Qwen2VLVisionBlock：**每个**VisionBlock**包含多头注意力机制和前馈网络**（MLP）**。它是视觉Transformer的核心模块。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tKZ7w8PNafWl"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xF5lQkzIaj2O"
      },
      "source": [
        "## **1.3 语言模型**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QYMwgDmOaxaN"
      },
      "source": [
        "Qwen2-VL 的语言模型部分主要处理文本输入，并生成对应的语言输出或进行推理。语言模型部分使用 Transformer 解码器架构。\n",
        "\n",
        "\n",
        "**语言模型配置**:**Qwen2VLConfig**\n",
        "\n",
        "**Qwen2VLConfig**类提供了整个语言模型的配置，它继承自**PretrainedConfig**，包括以下参数："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "18TfrjT6cBGC"
      },
      "outputs": [],
      "source": [
        "class Qwen2VLConfig(PretrainedConfig):\n",
        "    model_type = \"qwen2_vl\"\n",
        "    def __init__(self, vocab_size=152064, hidden_size=8192, intermediate_size=29568, num_hidden_layers=80, num_attention_heads=64, num_key_value_heads=8, hidden_act=\"silu\", max_position_embeddings=32768, initializer_range=0.02, rms_norm_eps=1e-05, use_cache=True, tie_word_embeddings=False, rope_theta=1000000.0, use_sliding_window=False, sliding_window=4096, max_window_layers=80, attention_dropout=0.0, vision_config=None, rope_scaling=None, **kwargs):\n",
        "        super().__init__(**kwargs)\n",
        "        self.vocab_size = vocab_size\n",
        "        self.hidden_size = hidden_size\n",
        "        self.intermediate_size = intermediate_size\n",
        "        self.num_hidden_layers = num_hidden_layers\n",
        "        self.num_attention_heads = num_attention_heads\n",
        "        self.num_key_value_heads = num_key_value_heads\n",
        "        self.hidden_act = hidden_act\n",
        "        self.max_position_embeddings = max_position_embeddings\n",
        "        self.initializer_range = initializer_range\n",
        "        self.rms_norm_eps = rms_norm_eps\n",
        "        self.use_cache = use_cache\n",
        "        self.tie_word_embeddings = tie_word_embeddings\n",
        "        self.rope_theta = rope_theta\n",
        "        self.use_sliding_window = use_sliding_window\n",
        "        self.sliding_window = sliding_window\n",
        "        self.max_window_layers = max_window_layers\n",
        "        self.attention_dropout = attention_dropout\n",
        "        self.vision_config = vision_config\n",
        "        self.rope_scaling = rope_scaling\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PKBoA-kwca21"
      },
      "source": [
        "**语言模型的实现**\n",
        "\n",
        "语言模型的实现基于 Transformer 解码器，使用多层自注意力（Self-Attention）和前馈网络（Feed-Forward Network）。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TT-_XeOvcJGq"
      },
      "outputs": [],
      "source": [
        "class Qwen2VLForConditionalGeneration(PreTrainedModel):\n",
        "    def __init__(self, config: Qwen2VLConfig):\n",
        "        super().__init__(config)\n",
        "        self.config = config\n",
        "\n",
        "        # 词嵌入层\n",
        "        self.embedding = nn.Embedding(config.vocab_size, config.hidden_size)\n",
        "\n",
        "        # Transformer 解码器\n",
        "        self.decoder = nn.ModuleList([Qwen2VLDecoderLayer(config) for _ in range(config.num_hidden_layers)])\n",
        "\n",
        "        # 输出层\n",
        "        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)\n",
        "\n",
        "    def forward(self, input_ids):\n",
        "        # 嵌入并处理位置编码\n",
        "        x = self.embedding(input_ids)\n",
        "\n",
        "        # 通过 Transformer 解码器\n",
        "        for layer in self.decoder:\n",
        "            x = layer(x)\n",
        "\n",
        "        # 输出生成的 logits\n",
        "        logits = self.lm_head(x)\n",
        "\n",
        "        return logits\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5_pbQsEYcm-F"
      },
      "source": [
        "**代码解析:**\n",
        "\n",
        "**self.embedding：**将文本 token 转换为词嵌入（embedding），这些嵌入表示文本中的语义。\n",
        "\n",
        "**Qwen2VLDecoderLayer：**每一层包含自注意力机制、前馈网络（MLP）和层归一化。它是解码器的核心模块。\n",
        "\n",
        "**lm_head：**输出层，将隐藏层的输出映射回词汇表大小，以生成每个位置的 token 概率分布。\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sVJ15k5tdBzF"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8Wv6JaMheAWP"
      },
      "source": [
        "## **1.4 多模态融合**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wA7Sp_gieIZL"
      },
      "source": [
        "Qwen2-VL 模型的独特之处在于，它能够处理并融合视觉和语言信息。视觉输入通过视觉编码器处理后与语言输入的特征进行融合，从而完成跨模态任务。\n",
        "\n",
        "**融合方式：**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xUCWjXmveMVD"
      },
      "outputs": [],
      "source": [
        "class Qwen2VLForConditionalGeneration(PreTrainedModel):\n",
        "    def forward(self, input_ids, visual_features):\n",
        "        # 将视觉特征和文本输入拼接\n",
        "        x = torch.cat([input_ids, visual_features], dim=1)\n",
        "\n",
        "        # 通过 Transformer 解码器处理\n",
        "        for layer in self.decoder:\n",
        "            x = layer(x)\n",
        "\n",
        "        # 输出生成的 logits\n",
        "        logits = self.lm_head(x)\n",
        "\n",
        "        return logits\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fBelQ_pXeRE-"
      },
      "source": [
        "**代码解析:**\n",
        "\n",
        "**visual_features：**这是经过视觉编码器处理后的图像特征。它们与文本输入的 input_ids 一起输入到解码器中。\n",
        "\n",
        "**torch.cat：**视觉特征和文本输入拼接在一起，形成多模态输入，之后通过 Transformer 解码器进行处理。\n",
        "\n",
        "\n",
        "\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3.10.2 64-bit",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.10.2"
    },
    "vscode": {
      "interpreter": {
        "hash": "0540c6a2c8e8c705c383409f733c32546cf3677e540abc16a3fc3931d30f97fb"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
